Image processing device, image processing method, and image pickup device

ABSTRACT

Disclosed herein is an image processing device including a subband dividing section configured to perform subband division of image data of a color whose pixel positions are alternately shifted from each other, the image data being included in image data output from an image pickup element of a pixel arrangement in which the pixel positions of at least one color of three primary colors are alternately shifted from each other in one of a horizontal direction and a vertical direction, with pixels of two upper and lower lines adjacent to each other or pixels of two left and right columns adjacent to each other as a unit.

This is a division of application Ser. No. 12/965,194, filed Dec. 10,2010, which is entitled to the priority filing date of Japaneseapplication(s) 2009-288074, filed on Dec. 18, 2009, the entirety ofwhich is incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing device, an imageprocessing method, and an image pickup device suitably applied forefficient compression coding of RAW data obtained from image pickupelements of various kinds of arrangement systems, for example.

2. Description of the Related Art

An image pickup device using an image pickup element of a Bayerarrangement is generally known in the past. Such an image pickup elementcaptures image light of a subject via a color filter, and outputs animage signal according to the intensity of the image light. Then, asubsequent processing section subjects the image signal to predeterminedprocessing, whereby the image pickup device can display an image on aviewfinder or an external display device.

Japanese Patent Laid-Open No. 2002-247376 (hereinafter referred to asPatent Document 1) describes compression of RAW data (image data beforecolor interpolation) obtained from an image pickup element of the Bayerarrangement as it is by JPEG (joint photographic experts group) or thelike.

Japanese Patent Laid-Open No. 2003-125209 (hereinafter referred to asPatent Document 2) describes techniques for separately performing imagecompression of each of components of G1, G2, R, and B and a concreteexample using wavelet compression as methods for compressing RAW dataobtained from an image pickup element of the Bayer arrangement.

SUMMARY OF THE INVENTION

The pixel positions of green pixels are alternately, shifted from eachother in a horizontal direction and a vertical direction in the past.Thus, the green pixels are divided into two components of G1 and G2 sothat the pixel positions are not shifted from each other, and thencompressed. This compression is performed after pixels originally havingstrong correlation therebetween as one image are separated into separatepictures by sub-sampling. Thus, the correlation between the separatedimages cannot be used, and compression efficiency is decreased.

A wavelet transform in particular can achieve a very high compressionefficiency by subband division of an entire picture. However, theexisting system separates the entire picture into separate pictures, andthus does not exert the inherently high compression efficiency of thewavelet transform.

In addition, images of different resolutions can be obtained from onecompression code by repeating subband division using a wavelettransform. When an image of half of a certain resolution of an image isdisplayed on a viewfinder or the like by the techniques described inPatent Document 2, for example, only one of two divided green images isused. In this case, a simple “pixel discrete reduction” is performed,and the image displayed on the viewfinder is affected by aliasing noise,for example. Thus, the advantages of using a wavelet transform cannot befully enjoyed.

In addition, the existing techniques relate to a method of compressingRAW data obtained from an image pickup element of a Bayer arrangement,but do not relate to techniques for compressing RAW data obtained froman image pickup element of a double density Bayer arrangement or athree-panel image pickup element system having pixels arranged in anoblique direction.

Thus, different compression systems need to be used for different pixelarrangements of image pickup elements, and hardware cannot be shared.

In addition, Patent Document 1 discloses a technique for performingcompression and recording without color separation into RGB full pixels.This technique compresses image data as it is without an increase indata rate due to color separation, and thus has an advantage of beingable to control the data rate. However, details on how to compress imagedata are not clear. If image data is literally compressed as one imageas it is in a Bayer arrangement, the level of each of RGB pixels differsin many natural images. Then, because adjacent pixels are differentlypainted into RGB, very large high-frequency components occur, andcompression efficiency is not increased. In other words, very highcompression noise is expected.

It is desirable to perform efficient compression coding of RAW dataobtained from an image pickup element of a pixel arrangement in whichpixel positions of at least one color of three primary colors arealternately shifted from each other in a horizontal direction or avertical direction.

The present invention performs subband division of image data of a colorwhose pixel positions are alternately shifted from each other, whichimage data is included in image data output from an image pickup elementof a pixel arrangement in which pixel positions of at least one color ofthree primary colors are alternately shifted from each other in ahorizontal direction or a vertical direction. This subband division isperformed with pixels of two upper and lower lines adjacent to eachother or pixels of two left and right columns adjacent each other as aunit.

According to the embodiment of the present invention, even when pixelpositions are alternately shifted from each other in a horizontal orvertical direction, subband division is performed with pixels of twoupper and lower lines adjacent to each other or pixels of two left andright columns adjacent each other as a unit. Thus, by performingcompression coding with the pixel positions remaining shifted from eachother, efficient compression coding can be performed.

According to the embodiment of the present invention, it is possible toperform efficient compression coding of image data (RAW data) obtainedfrom an image pickup element of a pixel arrangement in which pixelpositions of at least one color of three primary colors are alternatelyshifted from each other in a horizontal direction or a verticaldirection.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing an image pickup device according to anembodiment of the present invention;

FIG. 2 is a diagram showing an example of pixel arrangements of imagepickup elements;

FIG. 3 is a block diagram of a compression and decompression processingsection at a time of compression coding;

FIG. 4 is a diagram illustrating a division level of a wavelettransform;

FIGS. 5A and 5B are diagrams illustrating division levels of a wavelettransform;

FIG. 6 is a block diagram of the compression and decompressionprocessing section at a time of compression decoding;

FIGS. 7A, 7B, and 7C are diagrams of the related art;

FIGS. 8A, 8B, 8C, and 8D are diagrams in which the related art isapplied to a double density Bayer arrangement;

FIG. 9 is a diagram of processing of a compression and decompression I/Fsection at a time of a wavelet transform;

FIG. 10 is a diagram of processing of the compression and decompressionI/F section at a time of an inverse transform;

FIGS. 11A, 11B, 11C, and 11D are diagrams of wavelet transformprocessing of the compression and decompression I/F section for thedouble density Bayer arrangement;

FIGS. 12A, 12B, 12C, and 12D are diagrams of another example of wavelettransform processing of the compression and decompression I/F sectionfor the double density Bayer arrangement;

FIGS. 13A, 13B, 13C, and 13D are diagrams showing pixel barycentricpositions of subband images after a wavelet transform;

FIGS. 14A, 14B, 14C, and 14D are diagrams of wavelet transformprocessing of the compression and decompression I/F section for anoblique arrangement three-panel system;

FIGS. 15A, 15B, 15C, and 15D are diagrams of wavelet transformprocessing of the compression and decompression I/F section for a Bayerarrangement;

FIG. 16 is a diagram showing outputs of the compression anddecompression I/F section for various kinds of pixel arrangements;

FIG. 17 is a diagram of concrete processing at a time of a wavelettransform of the compression and decompression I/F section for an RGBfull pixel system;

FIG. 18 is a diagram of concrete processing at a time of an inversetransform of the compression and decompression I/F section for the RGBfull pixel system;

FIG. 19 is a diagram of concrete processing at a time of a wavelettransform of the compression and decompression I/F section for thedouble density Bayer arrangement;

FIG. 20 is a diagram of concrete processing at a time of an inversetransform of the compression and decompression I/F section for thedouble density Bayer arrangement;

FIG. 21 is a diagram of concrete processing at a time of a wavelettransform of the compression and decompression I/F section for theoblique arrangement three-panel system;

FIG. 22 is a diagram of concrete processing at a time of an inversetransform of the compression and decompression I/F section for theoblique arrangement three-panel system;

FIG. 23 is a diagram of concrete processing at a time of a wavelettransform of the compression and decompression I/F section for the Bayerarrangement;

FIG. 24 is a diagram of concrete processing at a time of an inversetransform of the compression and decompression I/F section for the Bayerarrangement;

FIGS. 25A, 25B, 25C, and 25D are diagrams showing an example in whichthe compression and decompression I/F section performs a Haar transformfor the double density Bayer arrangement;

FIGS. 26A, 26B, 26C, and 26D are diagrams showing an example in whichthe compression and decompression I/F section performs a Haar transformfor the oblique arrangement three-panel system; and

FIGS. 27A, 27B, 27C, and 27D are diagrams showing an example in whichthe compression and decompression I/F section performs a Haar transformfor the Bayer arrangement.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The best mode for carrying out the invention (which best mode willhereinafter be referred to as embodiments) will hereinafter bedescribed. Incidentally, description will be made in the followingorder.

1. First Embodiment (Control of Compression Coding or Decoding: ExampleUsing Wavelet Transform)

2. Second Embodiment (Control of Compression Coding or Decoding: ExampleUsing Haar Transform)

3. Examples of Modification

<1. First Embodiment>

[Example of Compression-Coding or Decoding Image Using WaveletTransform]

A first embodiment of the present invention will hereinafter bedescribed with reference to FIGS. 1 to 24.

In the embodiment below, description will be made of an example appliedto an image pickup device 10 that efficiently compression-codes each ofthree R/G/B components in any RAW data obtained from an image pickupelement of a Bayer arrangement without RGB full pixels, an image pickupelement of a double density Bayer arrangement without RGB full pixels,or an image pickup system using three image pickup elements havingpixels arranged in an oblique direction without RGB full pixels. Threecomponent signals including elements of RGB pixels will hereinafter beabbreviated to “three R/G/B components.”

The image pickup device 10 according to the present example can realizecompression coding without decreasing compression efficiency even whenpixel positions are alternately shifted from each other in a horizontaldirection or a vertical direction because of absence of real pixels, byperforming compression coding with the pixel positions remaining shiftedfrom each other.

In this case, a discretely reduced image in which pixel positions arealternately shifted from each other in a horizontal direction or avertical direction is averaged by being combined with a wavelettransform. Thus, a reduced subband image with half a resolution becomesan image having RGB full pixels.

That is, simple, so-called color separation (De-Bayer) processing by awavelet transform is performed, and display can be made on a monitor notshown in the figure which monitor is provided to the image pickup device10. This is because images of different resolutions can be obtained fromidentical image data when a wavelet transform is used. Even RAW dataobtained from an image pickup element that needs color separation suchas Bayer color separation or the like is subjected to a wavelettransform, and thus can be easily displayed on the monitor or the like.

When an image obtained from an image pickup element of RGB full pixelsis wavelet-transformed, each of RGB is subband-divided into four images,and the image is resolved into a total of 12 subband images of RGB. Whena compression coding method according to the present example is used,RAW data can be treated as differences in the number of subband imagesafter a wavelet transform. For example, RAW data obtained from a Bayerarrangement can be treated as four subband images, RAW data obtainedfrom a double density Bayer arrangement can be treated as eight subbandimages, and RAW data obtained from a three-panel image pickup elementsystem having pixels arranged in an oblique direction can be treated assix subband images. Thus, compression coding processing after a wavelettransform can be realized as common processing.

In addition, a high-resolution image such as a 4K image or the like hasa massive amount of data to be handled, and would therefore require someparallel processing when real-time processing is performed. However, ahigh-resolution image such as a 4K image or the like can be treated asdifferences in the number of subband images as described above by usingthe compression coding method according to the present example. Thus, anecessary number of processing blocks can be operated in parallel witheach other as processing after an image is subband-divided, andoperating speed can be reduced by an amount corresponding to a degree ofparallelism. In addition, as will be described later, same effects canbe obtained with not only a wavelet transform but also a Haar transformfor simple hardware. Incidentally, 4K denotes an example ofspecifications for high resolution such as 4096 samples×2160 lines. 2Kdenotes an example of specifications for lower resolution than 4 k suchas 2048 samples×1080 lines.

FIG. 1 shows an example of the image pickup device 10 handling a 4Kimage and a 2K image.

The image pickup device 10 is an example of an image pickup devicecapable of image compression/decoding according to an embodiment of thepresent invention.

A lens block 101 controls a diaphragm and a zoom, and forms an opticalimage on an image pickup element section 102.

The image pickup element section 102 converts the optical image inputfrom the lens block 101 into a digital video signal, and outputsrecording RAW data (D102). In the present example, provision can be madefor any of image pickup elements having pixel arrangements as shown inFIG. 2 as image pickup elements.

<RGB Full Pixel System>

-   -   An RGB three-panel system in which light is separated into RGB        using an optical prism    -   A single-panel system of such a structure as to have light        wavelength sensitivity in a direction of depth of a sensor    -   A single-panel system in which one pixel is painted differently        into three RGB stripes as in a liquid crystal TV        <Double Density Bayer Arrangement>

A normal Bayer arrangement is doubled in pixel density, and is disposedobliquely at 45°. Thus, full pixels are obtained for G, and R pixels andB pixels are arranged in such a pixel arrangement as to be spaced in anoblique direction (a thick frame in FIG. 2 indicates pixels in a normalBayer arrangement).

<Oblique Arrangement Three-Panel System>

-   -   A pixel arrangement having pixels arranged obliquely at 45° as        one image pickup element and assuming that pixel interpolation        is performed between two pixels or four pixels adjacent to each        other in a horizontal direction or a vertical direction to        interpolate pixels indicated by a dotted line circle in a thick        frame in FIG. 2 (pixels in a normal Bayer arrangement).    -   A system performing image pickup using three image pickup        elements in combination with an optical prism    -   Or a single-panel system having the above-described pixel        arrangement and having such a structure as to have light        wavelength sensitivity in a direction of depth of a sensor        <Bayer Arrangement>    -   A so-called normal Bayer arrangement

A camera signal processing section 103 performs predetermined processingon the recording RAW data (D102) output by the image pickup elementsection 102 of the Bayer arrangement or the like and reproduced RAW data(D105) read from a recording media section 108 and subjected todecompression processing to be described later. Specifically, the camerasignal processing section 103 creates a RGB full 4K image (so-calledcolor separation) so that the RGB full 4K image can be recognized as animage, makes camera image adjustment for a white balance, brightness andthe like, and outputs a recording and reproduced 4K image (D103) to amonitor-out section 104.

The monitor-out section 104 outputs the video signal of the 4K image toan external 4K monitor or the like.

A compression and decompression I/F section 105 resolves the recordingRAW data (D102) from the image pickup element of the Bayer arrangementor the like into subband images in a 2K band by a wavelet transform. Thecompression and decompression I/F section 105 functions as a subbanddividing section for subjecting image data of a color whose pixelpositions are alternately shifted from each other, which image data isincluded in image data output from an image pickup element of a pixelarrangement in which pixel positions of at least one color of threeprimary colors are alternately shifted from each other in a horizontaldirection or a vertical direction, to subband division with pixels oftwo upper and lower lines adjacent to each other or pixels of two leftand right columns adjacent to each other as a unit.

The compression and decompression I/F section 105 scans the pixels ofthe image data of the color whose pixel positions are alternatelyshifted from each other with two upper and lower lines adjacent to eachother as a unit, and performs a wavelet transform in the horizontaldirection. Alternatively, the compression and decompression I/F section105 scans the pixels of the image data of the color whose pixelpositions are alternately shifted from each other with two left andright columns adjacent to each other as a unit, and performs a wavelettransform in the vertical direction.

In the present example, the compression and decompression I/F section105 outputs the following subband images.

A subband image in the 2K band as a low-frequency component in both thehorizontal direction and the vertical direction is output as D105-LL.

A subband image in the 2K band as a high-frequency component in thehorizontal direction and a low-frequency component in the verticaldirection is output as D105-HL.

A subband image in the 2K band as a low-frequency component in thehorizontal direction and a high-frequency component in the verticaldirection is output as D105-LH.

A subband image in the 2K band as a high-frequency component in both thehorizontal direction and the vertical direction is output as D105-HH.

Depending on the pixel arrangement of the image pickup element section102, there is a case where not all the subband images HL/LH/HH in the 2Kband for each of RGB are output and some subband images are not output.Details will be described separately.

In addition, subband images in the 2K band (D105 -LL/HL/LH/HH) read fromthe recording media section 108 and subjected to decompressionprocessing are subjected to an inverse wavelet transform, and thenoutput as reproduced RAW data (D105). This will also be described indetail separately.

A compression and decompression processing section 106 image-compresseseach of the subband images in the 2K band (D105-LL/HL/LH/HH) using acompression coding system, and outputs the compressed subband images ascorresponding code streams (D106-LL/HL/LH/HH). Because the compressionand decompression I/F section 105 uses a wavelet transform, it is mostdesirable that the compression and decompression processing section 106adopt JPEG2000 or the like using the same wavelet transform. However,the compression and decompression processing section 106 may use anotherexisting image compression system.

In addition, the compression and decompression processing section 106decompresses each piece of subband image compressed data recorded in therecording media section 108, and outputs the decompressed data asreproduced subband images in the 2K band (D1 -LL/HL/LH/HH). At thistime, the compression and decompression processing section 106 functionsas a compression coding section for compression-coding image data outputfrom the compression and decompression I/F section 105 in parallel ineach band divided by the compression and decompression I/F section 105and in each of the three primary colors.

A recording media interface section 107 makes fast access to recordingmedia, and interfaces to read and write compressed image data.

The recording media section 108 is recording media for recording andreproducing compressed image data. A nonvolatile memory such as a flashmemory or the like is applied to the recording media section 108.

A viewfinder signal processing section 109 is an example of a displaysignal outputting section for outputting an input image as a displaysignal for display on a display system. The subband image D105-LL of thesubband images in the 2K band is a subband image of low frequencies inboth the horizontal direction and the vertical direction, and cantherefore be monitored as an RGB full image of 2K size. In this manner,camera image adjustment for a white balance, brightness and the like canbe made in 2K as in the camera signal processing section 103. Further,the viewfinder signal processing section 109 generates settinginformation for photographing, performs a peaking process forfacilitating focusing and the like, and outputs a recording andreproduced 2K image (D109) to a viewfinder section 110.

The viewfinder section 110 displays the recording and reproduced 2Kimage (D109) from the viewfinder signal processing section 109.

A system controlling section 111 has a controlling software program, andcontrols the whole of the image pickup device 10 according to theprogram. In addition, in response to input from an operating section112, the system controlling section 111 connects each block to a databus, exchanges data, and controls settings and a state forphotographing.

The operating section 112 receives an operation on the image pickupdevice 10, and transmits the operation as an electric signal to thesystem controlling section 111.

As already described, the compression and decompression I/F section 105uses a wavelet transform. Thus, description will first be made of thecompression and decompression processing section 106 performingcompression and decompression using the same wavelet transform.

FIG. 3 is a detailed block diagram of the compression and decompressionprocessing section 106 at a time of compression coding.

A wavelet transform section 32 subjects the subband image D105-LL in the2K band to a wavelet transform, and then outputs a wavelet transformcoefficient D32-LL.

The wavelet transform section 32 is generally realized by a filter bankincluding a low-pass filter and a high-pass filter. Incidentally, adigital filter generally has an impulse response of a length of aplurality of taps (filter coefficients), and thus an input image orcoefficients sufficient for performing filtering needs to be buffered inadvance. Also in a case of performing a wavelet transform in multiplestages, wavelet transform coefficients generated in a preceding stagewhich coefficients are necessary to perform filtering need to bebuffered in advance.

Subband images generated by a wavelet transform will be described below.

FIG. 4 shows an example of subband images. Generally, in this wavelettransform, as shown in FIG. 4, low-frequency components are repeatedlytransformed and divided. This is because much of the energy of an imageconcentrates in the low-frequency components. This is clear also from afact that subband images are formed as a division level is advanced froma division level=1 shown in FIG. 5A to a division level=3 shown in FIG.5B.

The division level of a wavelet transform in FIG. 4 is 3, and a total of10 subband images are formed as a result of the wavelet transform. Inthis case, L and H in FIG. 4 denote low frequencies and highfrequencies, respectively, and a number in front of L or H indicates adivision level. Specifically, 1LH, for example, denotes a subband imageat a division level=1 which subband image has low frequencies in thehorizontal direction and has high frequencies in the vertical direction.

A return will be made to the description of FIG. 3. The wavelettransform coefficient D32-LL is next quantized by a quantizing section33. Then a quantized coefficient D33-LL is output. It suffices to usescalar quantization, which is also used in JPEG2000, for a quantizingsection in this case. As shown in (Equation 1) below, it suffices to seta value obtained by dividing a wavelet transform coefficient W by aquantization step size D as the value of a quantized coefficient q.q=W/Δ  (Equation 1)

The quantization step size Δ37-LL is supplied from a code amountmeasuring section 37 to be described later.

The quantized coefficient D33-LL is next output to an entropy codingsection 35. The entropy coding section 35 compresses the quantizedcoefficient D33-LL using an arbitrary information source compressingsection. It suffices to adopt, as an entropy coding section, commonlyused Huffman coding (a system adopted in MPEG and JPEG, which systemgenerates code referring to a Huffman coding table created according tothe frequency of occurrence of symbols appearing in data in advance) orarithmetic coding (a system adopted in H.264 and JPEG2000). In addition,at that time, though not described in detail herein, the quantizedcoefficient may be combined with EBCOT (Embedded Block Coding withOptimal Truncation) as entropy coding in bit plane units as in JPEG2000.

A result coded by the entropy coding section 35 is output as a codedcode stream D35-LL to become the output D106-LL of the compression anddecompression processing section 106, and is also input to the codeamount measuring section 37.

The code amount measuring section 37 compares amounts of code of thecoded code stream D35-LL within one frame with a target code amountD36-LL supplied from a controlling section 36 while accumulating theamounts of code of the coded code stream D35-LL within one frame. Whenthe accumulation of the amounts of code of the coded code stream D35-LLwithin one frame is likely to exceed the target code amount, the codeamount measuring section 37 changes the quantization step size D37-LLfor the quantizing section 33 to a size larger by one step.

Conversely, when the accumulation of the amounts of code of the codedcode stream D35-LL within one frame is likely to be less than the targetcode amount, the code amount measuring section 37 changes thequantization step size D37-LL for the quantizing section 33 to a sizesmaller by one step.

The above is the description of operation of the compression anddecompression processing section 106 at the time of compression coding.However, not only the subband image LL component in the 2K band but alsothe HL component, the LH component, and the HH component are input tothe compression and decompression processing section 106.

The wavelet transform section 32 may also be used for the HL/LH/HHcomponent as for the LL component to raise the division level. However,in the example of FIG. 3, a wavelet transform at a first level hasalready been performed in the compression and decompression I/F section105. Thus, a mode of repeatedly transforming and dividing only thesubband image D105-LL of the low-frequency components is adoptedaccording to a common wavelet transform utilizing a property such thatmuch of the energy of the image shown in FIG. 4 concentrates in thelow-frequency components.

Incidentally, there is a case where for example hardware can realizeprocessing from the wavelet transform to entropy coding as one circuit.In this case, no limitation is imposed on forming the compression anddecompression processing section 106 using four existing circuits inparallel with each other without taking the trouble to develop a newcircuit not performing a wavelet transform.

In the example of FIG. 3, only quantization and entropy coding areperformed and the wavelet division level is not raised for the subbandimages D105-HL, D105 -LH, and D105-HH in the 2K band other than the LLcomponent.

Then, the subband images D105-HL, D105-LH, and D105-HH in the 2K bandother than the LL component are output as coded code streams D35-HL,D35-LH, and D35-HH, respectively. Then, the coded code streams D35-HL,D35-LH, and D35-HH become the outputs D106-HL, D106-LH, and D106 -HH ofthe compression and decompression processing section 106. In addition,code amount measuring sections 37 compare amounts of code of therespective coded code streams within one frame with target code amountsD36-HL, D36-LH, and D36-HH supplied from the controlling section 36while accumulating the amounts of code of the respective coded codestreams within one frame. As for the LL component, when the amounts ofcode of the respective coded code streams within one frame are likely toexceed the target code amounts, the code amount measuring sections 37change quantization step sizes for quantizing sections 33 to a sizelarger by one step. Conversely, when the accumulations of the amounts ofcode of the respective coded code streams within one frame are likely tobe less than the target code amounts, the code amount measuring sections37 change the quantization step sizes for the quantizing sections 33 toa size smaller by one step.

The amounts of code for the respective subband images in the 2K band arecontrolled as described above.

In the above description, the controlling section 36 sets the targetcode amounts for the respective subband images in the 2K band in therespective corresponding code amount measuring sections 37 in advance.However, for example, information on code amount accumulation conditionsmay be sent from each code amount measuring section 37 to thecontrolling section 36, and the target code amount for each subband maybe accommodated and changed according to the code amount accumulationconditions of each of the code amount measuring sections 37. Because anatural image can include a large amount of low-frequency components orconversely include a large amount of high-frequency components, optimumcode amount control can be performed according to the properties of theimage.

The above is the description of operation at the time of compressioncoding.

Next, FIG. 6 is a detailed block diagram of the compression anddecompression processing section 106 at a time of compression decoding,and operation at the time of compression decoding will be described.

Entropy decoding sections 38 supplied with coded code streams D106-LL,D106-HL, D106-LH, and D106-HH perform decoding according to a sectioncorresponding to the entropy coding described with reference to FIG. 3.Quantized coefficients D38-LL, D38-HL, D38-LH, and D38-HH are generatedas a result of the entropy decoding.

The quantized coefficients D38-LL, D38-HL, D38-LH, and D38-HH areconverted from the quantized coefficients D38-LL, D38-HL, D38-LH, andD38-HH to wavelet transform coefficients D39-LL, D39-HL, D39-LH, andD39-HH in dequantizing sections 39. A dequantizing section in this caseperforms an operation as the inverse of (Equation 1), and can beexpressed by the following (Equation 2):W=q×Δ  (Equation 2)(W is a wavelet transform coefficient, q is a quantized coefficient, andA is quantization step size.)

The wavelet transform coefficient D39-LL is restored to the subbandimage D105-LL in the 2K band of the LL component in an inverse wavelettransform section 40. Then the subband image D105-LL is output.

It is clear that when the subband images D105-HL, D105-LH, and D105-HHin the 2K band other than the LL component are also subjected to waveletre-division at the time of coding, it suffices to provide an inversewavelet transform section 40 for each subband to restore the wavelettransform coefficients D39-HL, D39-LH, and D39-HH to the respectivesubband images in the 2K band.

The above is the description of operation at the time of compressiondecoding.

The problems of the techniques described in existing Patent Document 2will be summed up in the following with reference to FIGS. 7A to 7C.

FIG. 7A shows an example of a 4K Bayer RAW image using a normal Bayerarrangement.

Though there are 4K pixels, the 4K pixels are painted differently intoRGB in the normal Bayer arrangement as shown in FIG. 7A, and only lightwavelength components corresponding to RGB color filters are convertedfrom light into an electric signal in the respective pixels. In thenormal Bayer arrangement, G is arranged in a checkered (checkered flag)pattern.

FIG. 7B shows an example of four color components of R pixels, G1pixels, G2 pixels, and B pixels obtained by separating the pixels of the4K Bayer RAW image.

In the techniques described in Patent Document 2, the 4K Bayer RAW imagedescribed above is perceived as the four color components of R pixels,G1 pixels, G2 pixels, and B pixels.

FIG. 7C shows an example in which the four color components shown inFIG. 7B are collected for different pixels.

In the techniques described in Patent Document 2, the pixels included inthe color components shown in FIG. 7B are collected to form a 2K Rimage, a 2K G1 image, a 2K G2 image, and a 2K B image, and the 2K Rimage, the 2K G1 image, the 2K G2 image, and the 2K B image are eachsubjected to compression coding using a wavelet transform and thenrecorded.

The G1 pixels and the G2 pixels are originally adjacent to each other inan oblique direction, and it is known that the G1 pixels and the G2pixels have very high correlation therebetween. Nevertheless, thetechniques described in Patent Document 2 take the trouble tocompression-code the G1 image and the G2 image as if the G1 image andthe G2 image were two independent images (or two color components), thusdecreasing compression efficiency. Although the wavelet transform inparticular achieves high compression efficiency in that an entire screenis converted into subbands, it is difficult to say that a combination ofthe existing techniques and the wavelet transform enjoys a highcompression ratio inherent in the wavelet transform.

In addition, as shown in FIG. 7B, “discrete reduction” due tosub-sampling of the G1 image and the G2 image is performed. It is thusobvious that when only the G1 image or the G2 image is displayed on a 2Kmonitor, aliasing occurs according to a sampling theorem. Accordingly,both the G1 image and the G2 image may be used to generate a 2K averageimage by signal processing and then display the 2K average image on themonitor, but it is necessary to access twice the data of 2K. However,the wavelet transform is subband division, and is thus intrinsicallycapable of enabling display on the monitor when only a 2K low-frequencycomponent is accessed. Also in this respect, it is difficult to say thatthe techniques described in Patent Document 2 fully enjoy the advantagesof the wavelet transform.

FIGS. 8A to 8D show an example of the existing techniques being appliedto the double density Bayer arrangement.

In FIG. 8A, the normal Bayer arrangement is doubled in pixel density,and pixels are obliquely arranged at 45°.

In this arrangement, a G image is 4K full pixels and an R image and a Bimage are obliquely spaced but provide a 4K resolution in the horizontaldirection and the vertical direction. The pixel density of thearrangement is twice that of the normal Bayer arrangement, and thus thearrangement will be referred to as a “double density Bayer arrangement”in the following description.

Consideration will now be given to a case of compressing a 4K RAW imageusing this double density Bayer arrangement according to the techniquesdescribed in Patent Document 2. In this case, unlike the Bayerarrangement, all 4K pixels are provided for G, while the R image and theB image are in a checkered pattern, as shown in FIG. 8B. Thus, as shownin FIG. 8C, the 4K RAW image is treated as five color components of R1pixels, R2 pixels, G pixels (4K resolution), B1 pixels, and B2 pixels.Then, the pixels are collected as shown in FIG. 8D to form a 2K R1image, a 2K R2 image, a 4K G image, a 2K B1 image, and a 2K B2 image.Each image is subjected to compression coding using a wavelet transform,and then recorded.

While only the G image is subjected to sub-sampling separation into twoscreens and then compressed in the Bayer arrangement, both the R imageand the B image are subjected to sub-sampling separation into twoscreens in the double density Bayer arrangement. Further, in the Bayerarrangement, the G image of high pixel density is separated into twoscreens and thus originally has high pixel density even when thecompression efficiency of the G image is decreased, and a balance can beattained with the compressed image quality of the R image and the Bimage of low pixel density.

In the double density Bayer arrangement, on the other hand, both the Rimage and the B image of low pixel density are further subjected tosub-sampling separation into two screens, and thus have low pixeldensity and are decreased in compression efficiency as well, and thebalance with the compressed image quality of the G image is furtherdisturbed.

In addition, as described in relation to the G image of the Bayerarrangement, there is a fear of occurrence of aliasing when thesub-sampled R image and the sub-sampled B image are displayed on a 2Kmonitor as they are.

On the other hand, the image pickup device 10 according to the presentexample can be expected to provide a great improvement utilizing theinherent features of the wavelet transform in relation to RAW datacompression for the R image and the B image in the double density Bayerarrangement in particular. Further, pixel arrangements from the normalBayer arrangement to RGB full pixels can be dealt with by a same method.

An example of processing at the time of a wavelet transform of thecompression and decompression I/F section 105 will next be describedwith reference to FIG. 9.

The compression and decompression I/F section 105 can maximize theeffect thereof when using a wavelet transform. Thus, though thecompression and decompression I/F section 105 has the same basicconfiguration as the wavelet transform section 32 in the compression anddecompression processing section 106, the processing of the compressionand decompression I/F section 105 will be described anew with referenceto FIG. 9.

Recording RAW data D102 is input as an input image of the compressionand decompression I/F section 105. In this case, supposing that 4K RGBfull pixels shown in FIG. 2 are input, three R/G/B components of4096×2160 size are subjected to exactly the same processing.

First, the input image is subjected to a wavelet transform in thehorizontal direction. The wavelet transform includes a low-pass filter(LPF) and a high-pass filter (HPF), which each perform 2:1 down sampling(indicated by a downward arrow and a FIG. 2 in FIG. 9). As a result, alow-frequency subband image (L) and a high-frequency subband image (H)of 2048×2160 size halved in the horizontal direction are obtained.

Next, the 2048×2160 low-frequency subband image (L) is subjected to awavelet transform in the vertical direction. The wavelet transformperforms 2:1 down sampling via each of a low-pass filter (LPF) and ahigh-pass filter (HPF). As a result of this processing, a low-frequencysubband image (L, L) D105-LL and a high-frequency subband image (L, H)D105-LH of 2048×1080 size halved in the vertical direction are obtained.

In addition, the 2048×2160 high-frequency subband image (H) is subjectedto a wavelet transform in the vertical direction. The wavelet transformperforms 2:1 down sampling via each of a low-pass filter (LPF) and ahigh-pass filter (HPF). As a result of this processing, a low-frequencysubband image (H, L) D105-HL and a high-frequency subband image (H, H)D105-HH of 2048×1080 size halved in the vertical direction are obtained.

As described above, the compression and decompression I/F section 105subjects the three R/G/B components of 4096×2160 size to a wavelettransform, and obtains the following three components:

-   -   The subband image (L, L) D105-LL of 2048×1080 size passed        through the low-pass filters in both the horizontal direction        and the vertical direction is three R/G/B components    -   The subband image (L, H) D105-LH of 2048×1080 size passed        through the low-pass filter in the horizontal direction and the        high-pass filter in the vertical direction is three R/G/B        components    -   The subband image (H, L) D105-HL of 2048×1080 size passed        through the high-pass filter in the horizontal direction and the        low-pass filter in the vertical direction is three R/G/B        components    -   The subband image (H, H) D105-HH of 2048×1080 size passed        through the high-pass filters in both the horizontal direction        and the vertical direction is three R/G/B components

Thus, the four kinds of subband images of 2048×1080 size are convertedinto a total of 12 subband images of 2048×1080 size for three R/G/Bcomponents.

The processing of the compression and decompression I/F section 105 atthe time of an inverse wavelet transform will next be described withreference to FIG. 10.

Subband images (D105-LL/HL/LH/HH) of 2048×1080 size read from therecording media section 108 and subjected to decompression processingare each input for three R/G/B components as an input image of thecompression and decompression I/F section 105.

The subband image (L, L) D105-LL of 2048×1080 size passed through alow-pass filter in both the horizontal direction and the verticaldirection and the subband image (L, H) D105-LH of 2048×1080 size passedthrough a low-pass filter in the horizontal direction and a high-passfilter in the vertical direction are subjected to an inverse wavelettransform in the vertical direction. The inverse wavelet transformsubjects the subband image D105-LL to 1:2 up sampling (indicated by anupward arrow and a FIG. 2 in FIG. 10) and then passes the subband imageD105-LL through a low-pass filter (LPF), and subjects the subband imageD105-LH to 1:2 up sampling and then passes the subband image D105-LHthrough a high-pass filter (HPF). Then, both the subband images aresynthesized to thereby obtain a low-frequency subband image (L) of2048×2160 size doubled in the vertical direction.

In addition, the subband image (H, L) D105-HL of 2048×1080 size passedthrough a high-pass filter in the horizontal direction and a low-passfilter in the vertical direction and the subband image (H, H) D105-HH of2048×1080 size passed through a high-pass filter in both the horizontaldirection and the vertical direction are subjected to an inverse wavelettransform in the vertical direction. The inverse wavelet transformsubjects the subband image D105-HL to 1:2 up sampling and then passesthe subband image D105-HL through a low-pass filter (LPF), and subjectsthe subband image D105-HH to 1:2 up sampling and then passes the subbandimage D105-HH through a high-pass filter (HPF). Then, both the subbandimages are synthesized to thereby obtain a high-frequency subband image(H) of 2048×2160 size doubled in the vertical direction.

Next, the low-frequency subband image (L) of 2048×2160 size and thehigh-frequency subband image (H) of 2048×2160 size resulting from theinverse wavelet transforms in the vertical direction are subjected to aninverse wavelet transform in the horizontal direction. The inversewavelet transform subjects the low-frequency subband image (L) to 1:2 upsampling and then passes the low-frequency subband image (L) through alow-pass filter (LPF), and subjects the high-frequency subband image (H)to 1:2 up sampling and then passes the high-frequency subband image (H)through a high-pass filter (HPF). Then, both the subband images aresynthesized to thereby obtain reproduced RAW data D105 of 4096×2160 sizealso doubled in the horizontal direction.

As described above, the processing of the compression and decompressionI/F section 105 at the time of the inverse wavelet transform isperformed using the subband images (D105-LL/HL/LH/HH) of 2048×1080 sizeread from the recording media section 108 and subjected to decompressionprocessing. In this processing, the inverse wavelet transform isperformed in the vertical direction and the horizontal direction usingthe subband images for three R/G/B components, and thereby the subbandimages are decoded into three R/G/B components of 4096×2160 size.

How the compression and decompression I/F section 105 in the presentexample performs processing for the double density Bayer arrangementshown in FIG. 2 will now be described with reference to FIGS. 11A to11D.

FIG. 11A shows the double density Bayer arrangement formed by doublingthe pixel density of the normal Bayer arrangement and obliquelyarranging the normal Bayer arrangement at 45°, as described above. Inthe double density Bayer arrangement, a G image is 4K full pixels, andan R image and a B image are obliquely spaced but provide a 4Kresolution in the horizontal direction and the vertical direction.

At this time, the image pickup element section 102 has the doubledensity Bayer arrangement as a pixel arrangement obtained by doublingthe pixel density of the Bayer arrangement and obliquely arranging theBayer arrangement at 45°.

The compression and decompression I/F section 105 resolves R and B imagedata of RGB image data output from the image pickup element section 102of the double density Bayer arrangement into R and B subband images byscanning pixels with two upper and lower lines adjacent to each other asa unit and performing a wavelet transform in the horizontal direction orby scanning pixels with two left and right columns adjacent to eachother as a unit and performing a wavelet transform in the verticaldirection. The compression and decompression I/F section 105 resolves Gimage data into G subband images by performing a wavelet transform inthe horizontal direction with pixels of one line as a unit andperforming a wavelet transform in the vertical direction with pixels ofone column as a unit.

Specifically, first making description for G, it is clear that exactlythe same processing as the processing described with reference to FIG. 9suffices because the G image has 4K full pixels.

Specifically, the compression and decompression I/F section 105 subjectsthe G image of 4096×2160 size to a wavelet transform in the horizontaldirection as shown in FIG. 11B. Then, a low-frequency subband G image(L) and a high-frequency subband G image (H) of 2048×2160 size halved inthe horizontal direction are obtained as shown in FIG. 11C. Taking a 5×3reversible wavelet filter (five taps for a filter on a low-pass side andthree taps for a filter on a high-pass side) defined in JPEG2000 as anexample, an area enclosed by a dotted line in the figure represents apixel range covered by the five-tap filter (filter on the low-pass side)with a pixel at a left end in the figure (a pixel provided with a star)as a center of the pixel range. This is performed for every two pixelsin the horizontal direction, thereby realizing 2:1 down sampling. Fordetails, reference should be made to JPEG2000 standards or the like.

Next, the low-frequency subband G image (L) and the high-frequencysubband G image (H) of 2048×2160 size halved in the horizontal directionare each subjected to a wavelet transform in the vertical direction asshown in FIG. 11C. Then, the following subband images are obtained, asshown in FIG. 11D.

A subband G image (LL) of 2048×1080 size passed through a low-passfilter in both the horizontal direction and the vertical direction.

A subband G image (LH) of 2048×1080 size passed through a low-passfilter in the horizontal direction and a high-pass filter in thevertical direction.

A subband G image (HL) of 2048×1080 size passed through a high-passfilter in the horizontal direction and a low-pass filter in the verticaldirection.

A subband G image (HH) of 2048×1080 size passed through a high-passfilter in both the horizontal direction and the vertical direction.

Thus, the G image forming the 4K double density Bayer RAW image isconverted into four kinds of subband G images of 2048×1080 size.

Consideration will now be given to the R image and the B image. The Rimage and the B image are in a checkered pattern. Therefore, when themethod described with reference to FIGS. 8A to 8D is used, pixel densityis low and compression efficiency is decreased as well, as describedabove. Thus a balance with the compressed image quality of the G imageis further disturbed.

The compression and decompression I/F section 105 in the present exampleaccordingly subjects the R image and the B image to a wavelet transformwith two upper and lower lines adjacent to each other as a unit.

Specifically, as shown in FIG. 11B, the compression and decompressionI/F section 105 scans pixels of the R image in the form of W with twoupper and lower lines adjacent to each other as a unit, and performs awavelet transform as if the pixels formed pixel data of one line.

In addition, the compression and decompression I/F section 105 scanspixels of the B image in the form of M with two upper and lower linesadjacent to each other as a unit, and performs a wavelet transform as ifthe pixels formed pixel data of one line.

When two upper and lower lines are set as a unit, one line can beconsidered to have 4096 real pixels. Thus, the R image and the B imagecan be treated as a 4096×1080 image.

This 4096×1080 R image is subjected to a wavelet transform in thehorizontal direction to be converted into:

a subband R image of 2048×1080 size passed through a low-pass filter;and a subband R image of 2048×1080 size passed through a high-passfilter.

The 4096×1080 B image is similarly subjected to a wavelet transform inthe horizontal direction to be converted into:

a subband B image of 2048×1080 size passed through a low-pass filter;and

a subband B image of 2048×1080 size passed through a high-pass filter.

At a point in time that the wavelet transforms in the horizontaldirection have been performed, subband images of 2048×1080 size arealready obtained, and the size coincides with the size of the subbandimages of G. Thus, no wavelet transform is performed in the verticaldirection.

This means that subband images of uniform 2048×1080 size are obtainedfor all of RGB, that 2048×1080 RGB full images are obtained, and thatsimple so-called color separation has been performed.

Thus, the subband images as output from the compression anddecompression I/F section 105 have the same size in the 4K full pixelsystem and the 4K double density Bayer arrangement, so that processingfrom the compression and decompression processing section 106 on downcan be made to be common processing. This will be described in detailseparately.

In FIG. 11D, the following description is used for distinction fromother systems to be described later and for description of the R imageand the B image having been subjected to only a wavelet transform in thehorizontal direction.

-   -   The subband R image of 2048×1080 size passed through the        low-pass filter is a subband R image (LL)    -   The subband R image of 2048×1080 size passed through the        high-pass filter is a subband R image (HL)    -   The subband B image of 2048×1080 size passed through the        low-pass filter is a subband B image (LL)    -   The subband B image of 2048×1080 size passed through the        high-pass filter is a subband B image (HL)

While description has been made supposing that the compression anddecompression I/F section 105 scans R pixels in the form of W and scansB pixels in the form of M, it is clear that the form of W and the formof M are inverted when starting pixel positions of the R pixels and theB pixels become different.

A method for solving the problems by thus performing a wavelet transformwith two upper and lower lines adjacent to each other as a unit for theR image and the B image of the double density Bayer arrangement has beendescribed with reference to FIGS. 11A to 11D. A method for solving theproblems by performing a wavelet transform with two left and rightpixels adjacent to each other as a unit for the R image and the B imageof the double density Bayer arrangement will be described as anothermethod with reference to FIGS. 12A to 12D.

First, as for the G image, because the G image has 4K full pixels,exactly the same processing as the processing described with referenceto FIGS. 11A to 11D is performed, and will be omitted here.

Though obvious from FIGS. 12A to 12D, processing for the R image and theB image of the double density Bayer arrangement will be described inorder.

At this time, the compression and decompression I/F section 105 scanspixels of the R and B image data of RGB image data output from the imagepickup element section 102 of the double density Bayer arrangement withtwo right and left columns adjacent to each other as a unit, andperforms a wavelet transform in the vertical direction. Then, for the Gimage data, the compression and decompression I/F section 105 performs awavelet transform in the horizontal direction with pixels of one line asa unit and a wavelet transform in the vertical direction with pixels ofone column as a unit.

Specifically, as shown in FIG. 12B, R pixels and B pixels are sent to awavelet transform in the vertical direction as they are without beingsubjected to a wavelet transform in the horizontal direction. At thistime, as shown in FIG. 12C, the compression and decompression I/Fsection 105 scans the R pixels in the form of a leftward facing W withtwo left and right pixels adjacent to each other as a unit, and performsa wavelet transform as if the pixels formed pixel data arranged in onecolumn.

In addition, the compression and decompression I/F section 105 scans theB pixels in the form of a rightward facing W with two left and rightpixels adjacent to each other as a unit, and performs a wavelettransform as if the pixels formed pixel data arranged in one column.

When two left and right pixels are set as a unit, the R image and the Bimage can be considered to have 2160 real pixels in, the verticaldirection. Thus, the R image and the B image can be treated as a2048×2160 image.

This 2048×2160 R image is subjected to a wavelet transform in thevertical direction. At this time, the compression and decompression I/Fsection 105 converts the 2048×2160 R image, into:

a subband R image of 2048×1080 size passed through a low-pass filter;and

a subband R image of 2048×1080 size passed through a high-pass filter.

The 2048×2160 B image is similarly subjected to a wavelet transform inthe vertical direction. At this time, the compression and decompressionI/F section 105 converts the 2048×2160 B image into:

a subband B image of 2048×1080 size passed through a low-pass filter;and a subband B image of 2048×1080 size passed through a high-passfilter.

At a point in time that the wavelet transforms in the vertical directionhave been performed, subband images of 2048×1080 size are obtained, andthe size coincides with the size of the subband images of G.

Thus, the subband images as output from the compression anddecompression I/F section 105 have the same size in the 4K full pixelsystem and the 4K double density Bayer arrangement, so that processingfrom the compression and decompression processing section 106 on downcan be made to be common processing. This will be described in detailseparately.

In FIG. 12D, the subband images are described as follows for distinctionfrom other systems to be described later and for description of the Rimage and the B image having been subjected to only a wavelet transformin the vertical direction.

-   -   The subband R image of 2048×1080 size passed through the        low-pass filter is a subband R image (LL)    -   The subband R image of 2048×1080 size passed through the        high-pass filter is a subband R image (LH)    -   The subband B image of 2048×1080 size passed through the        low-pass filter is a subband B image (LL)    -   The subband B image of 2048×1080 size passed through the        high-pass filter is a subband B image (LH)

While description has been made above with reference to FIGS. 12A to 12Dsupposing that the compression and decompression I/F section 105 scans Rpixels in the form of a leftward facing W and scans B pixels in the formof a rightward facing W, it is clear that the form of the leftwardfacing W and the form of the rightward facing W are reversed whenstarting pixel positions of the R pixels and the B pixels becomedifferent.

How wavelet transform processing is performed for the double densityBayer arrangement has been described above with reference to FIGS. 11Ato 11D and FIGS. 12A to 12D.

The pixel barycentric positions of subband images of 2048×1080 sizepassed through a low-pass filter after a wavelet transform will next bedescribed with reference to FIGS. 13A to 13D.

Description in the following will be made using (−1/8, 2/8, 6/8, 2/8,−1/8) as JPEG2000 5×3 reversible wavelet filter coefficients as anexample.

FIG. 13A shows an example in which a 5×3 reversible wavelet filter isapplied to a G image. First, five pixels in the horizontal direction aremultiplied by the filter coefficients of (−1/8, 2/8, 6/8, 2/8, −1/8).The third pixel is multiplied by 6/8, and the two left pixels and thetwo right pixels with the third pixel as a center are multiplied by thesymmetric coefficients. Thus, the barycenter of the pixels in thehorizontal direction coincides with the position of the third pixel.

Next, five pixels in the vertical direction are multiplied by the filtercoefficients of (−1/8, 2/8, 6/8, 2/8, −1/8 ). The third pixel ismultiplied by 6/8 , and the two upper pixels and the two lower pixelswith the third pixel as a center are multiplied by the symmetriccoefficients. Thus, the barycenter of the pixels in the verticaldirection also coincides with the position of the third pixel.

That is, the pixel barycentric position of the subband image of2048×1080 size obtained by passing the G image through a low-passwavelet filter is the position of the pixel (circle mark ∘ in FIG. 13A)multiplied by the filter coefficient of 6/8 in both the horizontaldirection and the vertical direction.

FIG. 13B shows an example in which a 5×3 reversible wavelet filter isapplied to an R image. As described with reference to FIGS. 11A to 11D,the compression and decompression I/F section 105 scans pixels of the Rimage in the form of W with two upper and lower lines adjacent to eachother as a unit, and subjects the pixels to a wavelet transform as ifthe pixels were pixel data of one line.

The five pixels are multiplied by the filter coefficients of (−1/8, 2/8,6/8, 2/8, −1/8). The third pixel is multiplied by 6/8, and the two leftpixels and the two right pixels with the third pixel as a center aremultiplied by the symmetric coefficients. Thus, the barycenter of thepixels in the horizontal direction coincides with the position of thethird pixel.

The R image is not subjected to a wavelet transform in the verticaldirection. Directing attention to the above wavelet coefficients in thehorizontal direction, the filter coefficients of (−1/8, 6/8, −1/8) areapplied to the three pixels on the upper line. It is thus shown that theweight of the coefficients for the three pixels on the upper line is−1/8+6/8−1/8=1/2Next, the filter coefficients of (2/8, 2/8) are applied to the twopixels on the lower line, and it is thus shown that the weight of thecoefficients for the two pixels on the lower line is2/8+2/8=1/2

It is shown from the above that the barycenter of the pixels in thevertical direction of the R image is exactly a middle position betweenthe two lines where the pixels are scanned in the form of W with the twoupper and lower lines adjacent to each other as a unit.

That is, the pixel barycentric position of the subband image of2048×1080 size obtained by passing the R image through a low-passwavelet filter is the position of the pixel multiplied by the filtercoefficient of 6/8 in the horizontal direction and the position betweenthe lines set as the two upper and lower lines adjacent to each other asa unit in the vertical direction (circle mark ∘ in FIG. 13B).

FIG. 13C shows an example in which a 5×3 reversible wavelet filter isapplied to a B image. As described with reference to FIGS. 11A to 11D,the compression and decompression I/F section 105 scans pixels of the Bimage in the form of M with two upper and lower lines adjacent to eachother as a unit, and subjects the pixels to a wavelet transform as ifthe pixels were pixel data of one line.

The five pixels are multiplied by the filter coefficients of (−1/8, 2/8,6/8, 2/8, −1/8). The third pixel is multiplied by 6/8, and the two leftpixels and the two right pixels with the third pixel as a center aremultiplied by the symmetric coefficients. Thus, the barycenter of thepixels in the horizontal direction coincides with the position of thethird pixel.

The B image is not subjected to a wavelet transform in the verticaldirection. Directing attention to the above wavelet coefficients in thehorizontal direction, the filter coefficients of (−1/8, 6/8, −1/8) areapplied to the three pixels on the lower line. It is thus shown that theweight of the coefficients for the three pixels on the lower line is−1/8+6/8−1/8=1/2

Next, the filter coefficients of (2/8, 2/8) are applied to the twopixels on the upper line, and it is thus shown that the weight of thecoefficients for the two pixels on the upper line is2/8+2/8=1/2

It is shown from the above that the barycenter of the pixels in thevertical direction of the B image is exactly a middle position betweenthe two lines where the pixels are scanned in the form of M with the twoupper and lower lines adjacent to each other as a unit.

That is, the pixel barycentric position of the subband image of2048×1080 size obtained by passing the B image through a low-passwavelet filter is the position of the pixel multiplied by the filtercoefficient of 6/8 in the horizontal direction and the position betweenthe lines set as the two upper and lower lines adjacent to each other asa unit in the vertical direction (circle mark ∘ in FIG. 13C).

Summarizing the above, the pixel barycentric positions of the subbandimages of 2048×1080 size passed through the low-pass wavelet filters areas shown in FIG. 13D. At this time, it is shown that the R image and theB image have the same position, whereas the G image has the sameposition in the vertical direction and is only slightly shifted by a 1/2of a pixel.

One of the features of subband division using a wavelet filter such asJPEG2000 and the like is that image sizes of 1/2^(n) resolution can beobtained from one compressed stream. This shows that also in the doubledensity Bayer arrangement, the pixel barycentric positions of RGB aresubstantially the same when only a low-frequency subband image of halfan image size in each of the horizontal direction and the verticaldirection is displayed. That is, it is shown that a low-frequencysubband image substantially free from a color shift can be displayedwithout the pixel barycentric positions being made to coincide with eachother.

Of course, the pixel barycentric positions of RGB perfectly coincidewith each other when the G image is corrected by 1/2 of a pixel in thehorizontal direction. Also in this case, because only the G image iscorrected by 1/2 of a pixel only in the horizontal direction, an effectis obtained in that a small hardware scale suffices.

In addition, when the R image and the B image are subjected to a wavelettransform only in the vertical direction as shown in FIGS. 12A to 12D,it is shown that when similar consideration to that of FIGS. 13A to 13Dis applied, though not shown in detail, the R image and the B image havethe same position, whereas the G image has the same position in thehorizontal direction and is only slightly shifted by 1/2 of a pixel inthe vertical direction.

The above is details of processing for the double density Bayerarrangement.

An example of compression coding processing for the oblique arrangementthree-panel system in FIG. 2 will next be described with reference toFIGS. 14A to 14D.

FIG. 14A shows a pixel arrangement having pixels arranged obliquely at45° as one image pickup element 102 and assuming that pixelinterpolation is performed between two horizontal or vertical pixels orfour horizontal or vertical pixels adjacent to each other to interpolatepixels indicated by dotted line circle marks in the figure.

At this time, the image pickup element section 102 is the obliquearrangement three-panel system as a pixel arrangement having pixelsarranged obliquely at 45° and assuming that interpolation is performedbetween pixels adjacent to each other in the horizontal direction or thevertical direction.

Description in the following will be made of a method for applying thepresent invention to a system performing image pickup using three imagepickup elements 102 having the above-described pixel arrangement incombination with an optical prism or a single-panel type image pickupsystem having the pixel arrangement and having such a structure as tohave light wavelength sensitivity in a direction of depth of a sensor.

First, because it is assumed that the pixels indicated by the dottedline circle marks in FIG. 14A are interpolated from surrounding pixels,the pixels indicated by the dotted line circle marks in FIG. 14A can beconsidered to be nonexistent as real pixels. That is, as shown in FIG.14B, the pixels indicated by hatching are considered to be nonexistent,and this pattern can therefore be considered to be the same as that of Ror B pixels in the double density Bayer arrangement described above withreference to FIGS. 11A to 11D.

A difference lies in that an R image, a B image, and a G image all havethe same pattern.

It therefore suffices to subject the R image, the B image, and the Gimage to a wavelet transform with two upper and lower lines adjacent toeach other as a unit.

At this time, the compression and decompression I/F section 105 scanspixels of all of RGB image data from the image pickup element section102 of the oblique arrangement three-panel system with two upper andlower lines adjacent to each other as a unit and performs a wavelettransform in the horizontal direction, or scans pixels of all of the RGBimage data from the image pickup element section 102 of the obliquearrangement three-panel system with two left and right columns adjacentto each other as a unit and performs a wavelet transform in the verticaldirection.

Specifically, as shown in FIG. 14B, the compression and decompressionI/F section 105 scans pixels in the form of W with two upper and lowerlines adjacent to each other as a unit, and performs a wavelet transformas if the pixels were pixel data of one line. Depending on the startingposition of the pixels, it is possible to scan the pixels in the form ofM with the two upper and lower lines adjacent to each other as a unit,and perform a wavelet transform as if the pixels were pixel data of oneline.

When two upper and lower lines are set as a unit, one line can beconsidered to have 4096 real pixels. Thus, the RGB images can be treatedas a 4096×1080 image.

The 4096×1080 images are subjected to a wavelet transform in thehorizontal direction, whereby the RGB images are each converted into:

a subband image of 2048×1080 size passed through a low-pass filter; and

a subband image of 2048×1080 size passed through a high-pass filter.

At a point in time that the wavelet transforms in the horizontaldirection have been performed, subband images of 2048×1080 size arealready obtained. Thus, no wavelet transform is performed in thevertical direction.

This means that subband images of uniform 2048×1080 size are obtainedfor all of RGB, that 2048×1080 RGB full images are obtained, and thatsimple so-called color separation has been performed.

Thus, the subband images as output from the compression anddecompression I/F section 105 have the same size in the 4K full pixelsystem and the 4K oblique arrangement three-panel system, so thatprocessing from the compression and decompression processing section 106on down can be made to be common processing. This will be described indetail separately.

In FIG. 14D, the subband images of 2048×1080 size are described asfollows for distinction from other systems to be described later and fordescription of the images having been subjected to only a wavelettransform in the horizontal direction.

-   -   The subband R image of 2048×1080 size passed through the        low-pass filter is a subband R image (LL)    -   The subband R image of 2048×1080 size passed through the        high-pass filter is a subband R image (HL)    -   The subband G image of 2048×1080 size passed through the        low-pass filter is a subband G image (LL)    -   The subband G image of 2048×1080 size passed through the        high-pass filter is a subband G image (HL)    -   The subband B image of 2048×1080 size passed through the        low-pass filter is a subband B image (LL)    -   The subband B image of 2048×1080 size passed through the        high-pass filter is a subband B image (HL)

While description has been made with reference to FIGS. 14A to 14D ofprocessing in which the images of the oblique arrangement three-panelsystem are subjected to a wavelet transform with two upper and lowerlines adjacent to each other as a unit, it is also possible as anothermethod to perform a wavelet transform with two left and right pixelsadjacent to each other as a unit. However, the processing is exactly thesame as processing described as the processing of the R image and the Bimage of the double density Bayer arrangement as described withreference to FIGS. 12A to 12D, and will be omitted here.

The pixel barycentric positions of subband images of 2048×1080 sizepassed through a low-pass filter after a wavelet transform in the doubledensity Bayer arrangement have been described with reference to FIGS.13A to 13D.

However, in the oblique arrangement three-panel system, pixel positionsare adjusted so as to coincide optically with each other. It is thusobvious that the pixel barycentric positions of subband images of2048×1080 size passed through the low-pass filters after the wavelettransform for RGB are the same, and therefore description thereof willbe omitted.

The above is details of processing for the oblique arrangementthree-panel system.

An example of processing for the normal Bayer arrangement shown in FIG.2 will next be described with reference to FIGS. 15A to 15D.

FIG. 15A shows a so-called normal Bayer arrangement.

At this time, the image pickup element section 102 has the Bayerarrangement.

In FIG. 15B, the Bayer arrangement is separated into each of RGB, andpositions where no real pixel is present are indicated by hatching. TheR image and the B image are already discretely reduced images of2048×1080 size in the separated state.

As is clear from FIG. 15B, the G image can be considered to be the sameas the R pixels or the B pixels of the double density Bayer arrangementas described with reference to FIGS. 11A to 11D.

It therefore suffices to subject the G image to a wavelet transform withtwo upper and lower lines adjacent to each other as a unit.

At this time, the compression and decompression I/F section 105 does notapply a wavelet transform to R and B image data of RGB image data fromthe image pickup element section 102 of the Bayer arrangement. Then, thecompression and decompression I/F section 105 scans pixels of the Gimage data with two upper and lower lines adjacent to each other as aunit and performs a wavelet transform in the horizontal direction, orscans pixels of the G image data with two left and right columnsadjacent to each other as a unit and performs a wavelet transform in thevertical direction.

Specifically, as shown in FIG. 15B, the compression and decompressionI/F section 105 scans pixels in the form of M with two upper and lowerlines adjacent to each other as a unit, and performs a wavelet transformas if the pixels were pixel data of one line.

Depending on the starting position of the pixels, it is possible to scanthe pixels in the form of W with the two upper and lower lines adjacentto each other as a unit, and perform a wavelet transform as if thepixels were pixel data of one line.

When two upper and lower lines are set as a unit, one line can beconsidered to have 4096 real pixels. Thus, the G image can be treated asa 4096×1080 image.

The 4096×1080 G image is subjected to a wavelet transform in thehorizontal direction to be converted into:

a subband G image of 2048×1080 size passed through a low-pass filter;and

a subband G image of 2048×1080 size passed through a high-pass filter.

At a point in time that the wavelet transforms in the horizontaldirection have been performed, subband images of 2048×1080 size arealready obtained. Thus, no wavelet transform is performed in thevertical direction.

This means that subband images of uniform 2048×1080 size are obtainedfor all of RGB though the R image and the B image are discretelyreduced, that 2048×1080 RGB full images are obtained, and that simpleso-called color separation has been performed.

Thus, the subband images as output from the compression anddecompression I/F section 105 have the same size in the 4K full pixelsystem and the 4K normal Bayer arrangement, so that processing from thecompression and decompression processing section 106 on down can be madeto be common processing. This will be described in detail separately.

In FIG. 15D, description is made as follows to make the R image and theB image common for the processing of other systems, though the R imageand the B image are discretely reduced, and, in relation to the G image,for distinction from the other systems to be described later and fordescription of the image having been subjected to only a wavelettransform in the horizontal direction.

-   -   The R image of 2048×1080 size is a subband R image (LL)    -   The subband G image of 2048×1080 size passed through the        low-pass filter is a subband G image (LL)    -   The subband G image of 2048×1080 size passed through the        high-pass filter is a subband G image (HL)    -   The B image of 2048×1080 size is a subband B image (LL)

While description has been made with reference to FIGS. 15A to 15D of amethod in which the G image of the normal Bayer arrangement is subjectedto a wavelet transform with two upper and lower lines adjacent to eachother as a unit, it is also possible as another method to perform awavelet transform with two left and right pixels adjacent to each otheras a unit. However, the method is exactly the same as processingdescribed as the processing of the R image and the B image of the doubledensity Bayer arrangement as described with reference to FIGS. 12A to12D, and will be omitted here.

The pixel barycentric positions of subband images of 2048×1080 sizepassed through a low-pass filter after a wavelet transform in the doubledensity Bayer arrangement have been described with reference to FIGS.13A to 13D. However, in the normal Bayer arrangement, the R image andthe B image are subjected to simple discrete reduction processing as inother inventions. Thus, pixel positions remain those of the Bayerarrangement as they are, and therefore description thereof will beomitted.

The above is details of processing for the normal Bayer arrangement.

Description has been made to show that the compression and decompressionI/F section 105 is thus compatible with any of image pickup elements ofvarious pixel arrangements as shown in FIG. 2.

Description will next be made to show that processing from thecompression and decompression processing section 106 on down can be madeto be common processing because subband images as output from thecompression and decompression I/F section 105 have 2048×1080 size withany of image pickup element sections 102 of various pixel arrangementsas shown in FIG. 2.

FIG. 16 shows a summary of subband images as output from the compressionand decompression I/F section 105 in correspondence with the imagepickup element sections 102 of the various pixel arrangements describedthus far.

Specifically, the subband images can be summarized as differences in thenumber of subbands of 2048×1080 size as follows. (In FIG. 16, subbandimages not output are represented by a broken line.)

RGB Full Pixel System

-   -   G: four subband components (LL/HL/LH/HH) of 2048×1080 size    -   B: four subband components (LL/HL/LH/HH) of 2048×1080 size    -   R: four subband components (LL/HL/LH/HH) of 2048×1080        size→twelve subband components of 2048×1080 size

Double Density Bayer Arrangement

-   -   G: four subband components (LL/HL/LH/HH) of 2048×1080 size    -   B: two subband components (LL/HL) of 2048×1080 size    -   R: two subband components (LL/HL) of 2048×1080 size→eight        subband components of 2048×1080 size

Oblique Arrangement Three-Panel System

-   -   G: two subband components (LL/HL) of 2048×1080 size    -   B: two subband components (LL/HL) of 2048×1080 size    -   R: two subband components (LL/HL) of 2048×1080 size→six subband        components of 2048×1080 size

Bayer Arrangement

-   -   G: two subband components (LL/HL) of 2048×1080 size    -   B: one subband component (LL) of 2048×1080 size    -   R: one subband component (LL) of 2048×1080 size→four subband        components of 2048×1080 size

In addition, in any of the pixel arrangements, a subband (LL) (subbandsenclosed by a thick frame in FIG. 16) is present for each of RGB images.Thus, as described above, simple color separation is made, and thesubbands can be recognized as an image.

Description will next be made of concrete processing methods of thecompression and decompression I/F section 105 for the image pickupelements of the various pixel arrangements.

FIG. 17 and FIG. 18 are diagrams of assistance in explaining aprocessing method corresponding to the RGB full pixel system (FIG. 17corresponds to processing at a time of a wavelet transform, and FIG. 18corresponds to processing at a time of an inverse transform).

FIG. 17 and FIG. 18 are the same as FIG. 9 and FIG. 10, and thereforerepeated description will be omitted here. As shown in FIG. 17, thecompression and decompression I/F section 105 performs same processingon all of an R image, a G image, and a B image of recording RAW data ofthe RGB full pixel system. At this time, the compression anddecompression I/F section 105 performs a wavelet transform in both thehorizontal direction and the vertical direction of the R image, the Gimage, and the B image, and outputs 12 components of subbands of2048×1080 size. In addition, as shown in FIG. 18, the compression anddecompression I/F section 105 performs the inverse transform of thewavelet transform, and restores the 12 components to reproduced RAW dataof the RGB full pixel system.

FIG. 19 and FIG. 20 are diagrams of assistance in explaining aprocessing method corresponding to an image pickup element of the doubledensity Bayer arrangement (FIG. 19 corresponds to processing at a timeof a wavelet transform, and FIG. 20 corresponds to processing at a timeof an inverse transform).

As shown in FIG. 19, the compression and decompression I/F section 105performs a wavelet transform in both the horizontal direction and thevertical direction of a G image in recording RAW data of the doubledensity Bayer arrangement. However, for an R image and a B image, thecompression and decompression I/F section 105 performs only a wavelettransform in the horizontal direction after scanning as shown in FIG.11B, bypasses vertical direction wavelet transform circuits, and outputsthe results as they are. In FIG. 19, the bypassed parts are indicated byhatching. Bypassed parts are similarly represented also in FIGS. 20 to24 in the following. The compression and decompression I/F section 105outputs eight components of subbands of 2048×1080 size. As shown in FIG.20, also in the inverse transform, for the R image and the B image, thecompression and decompression I/F section 105 bypasses verticaldirection wavelet synthesis circuits, and performs only a waveletsynthesis in the horizontal direction. The compression and decompressionI/F section 105 can restore the G image to reproduced RAW data of thedouble density Bayer arrangement by performing a wavelet synthesis inboth the horizontal direction and the vertical direction.

FIG. 21 and FIG. 22 are diagrams of assistance in explaining aprocessing method corresponding to the oblique arrangement three-panelsystem (FIG. 21 corresponds to processing at a time of a wavelettransform, and FIG. 22 corresponds to processing at a time of an inversetransform).

As shown in FIG. 21, the compression and decompression I/F section 105performs same processing on all of an R image, a G image, and a B imagein recording RAW data of the oblique arrangement three-panel system. Atthis time, the compression and decompression I/F section 105 performsscanning as shown in FIG. 14B, performs only a wavelet transform in thehorizontal direction of the R image, the G image, and the B image,bypasses the vertical direction wavelet transform circuits, and outputsthe results as they are. The compression and decompression I/F section105 outputs six components of subbands of 2048×1080 size. In addition,as shown in FIG. 22, also in the inverse transform, the compression anddecompression I/F section 105 performs same processing on all of the Rimage, the G image, and the B image. The compression and decompressionI/F section 105 bypasses the vertical direction wavelet synthesiscircuits, and performs only a wavelet synthesis in the horizontaldirection. The compression and decompression I/F section 105 restoresthe six components to reproduced RAW data of the oblique arrangementthree-panel system.

FIG. 23 and FIG. 24 are diagrams of assistance in explaining aprocessing method corresponding to an image pickup element of the normalBayer arrangement (FIG. 23 corresponds to processing at a time of awavelet transform, and FIG. 24 corresponds to processing at a time of aninverse transform).

As shown in FIG. 23, for a G image in recording RAW data of the normalBayer arrangement, the compression and decompression I/F section 105performs scanning as shown in FIG. 15B, and performs a wavelet transformonly in the horizontal direction. However, as for an R image and a Bimage, the compression and decompression I/F section 105 does notperform a wavelet transform in the horizontal direction either, butperforms bypass processing. The compression and decompression I/Fsection 105 bypasses the vertical direction wavelet transform circuitsfor each of the R image, the G image, and the B image, and outputs theresults as they are. Thus, the compression and decompression I/F section105 outputs four components of subbands of 2048×1080 size.

In addition, as shown in FIG. 24, also in the inverse transform, thecompression and decompression I/F section 105 bypasses the verticaldirection wavelet synthesis circuits for each of the R image, the Gimage, and the B image, and performs a wavelet synthesis in thehorizontal direction only for the G image. As for the R image and the Bimage, the compression and decompression I/F section 105 does notperform a wavelet synthesis in the horizontal direction either, butperforms bypass processing. Thus, the images can be restored toreproduced RAW data of the normal Bayer arrangement.

When the compression and decompression I/F section 105 thus bypasseswavelet circuits according to a kind of input recording RAW data,provision can be made for the image pickup elements of the various pixelarrangements. Thus, by only controlling the processing of thecompression and decompression I/F section 105, processing from thecompression and decompression processing section 106 on down can be madeto be common processing only with differences in the number ofcomponents of input subband images.

In addition, processing from the compression and decompressionprocessing section 106 on down can be performed as parallel processing.Specifically, in the compression and decompression processing section106, the circuits for the subband image D105-LL in FIG. 3, the circuitsfor the subband image D105-HL in FIG. 3, the circuits for the subbandimage D105-LH in FIG. 3, and the circuits for the subband image D105-HHin FIG. 3 are each provided to three parallel systems for RGB,respectively (twelve systems in total). It is thus possible to use allof the systems in the case of a 4K RGB full image, and use only a partof the systems in other cases. Thus, operating speed can be reduced byan amount corresponding to a degree of parallelism. That is, aconfiguration suitable for real-time processing can be adopted.

In the past, a system handling high-resolution images such as 4K imagesand the like inevitably handles a large amount of signal processing. Inaddition, from a viewpoint of characteristics such as S/N, sensitivityand the like of the image pickup element, it is difficult to realize thesystem unless the number of pixels is reduced in advance by using theso-called Bayer arrangement or the like and then RAW data is recorded.However, because not all pixels of RGB are present, compressionefficiency is not increased when compression recording is performedusing the RAW data as it is, as described in the section of thebackground art.

On the other hand, by using the compression and decompression I/Fsection 105 and the compression and decompression processing section 106according to the first embodiment described above, it is possible toconvert an entire screen, which is a feature of the wavelet transform,and utilize high compression efficiency inherent in the wavelettransform. A great improvement in compression efficiency utilizing theinherent features of the wavelet transform can be expected with respectto compression of RAW data of an R image and a B image in the doubledensity Bayer arrangement in particular.

In addition, pictures of different resolutions as a result of subbanddivision can be obtained from one compression code, and the pictures canbe used. This is because even simple color separation processing isperformed by a wavelet transform.

Thus, it is possible to construct a system with a very high degree ofhardware efficiency, which system is a low-resolution simple displaysystem but does not require a special color separation circuit or adown-converting circuit.

Further, the processing of the compression and decompression I/F section105 and the compression and decompression processing section 106 can beadapted to any pattern of the existing RGB full pixel system, the doubledensity Bayer arrangement, the oblique arrangement three-panel system,and the normal Bayer arrangement. Thus, an effect of being able toperform common hardware processing with “differences in the number ofsubband image components” is produced. This indicates that an advantageof being able to perform parallel hardware processing and realizereal-time processing in a system handling high-resolution images is alsoprovided.

<2. Second Embodiment>

[Example of Compression-Coding or Decoding Image Using Haar Transform]

An example applied to an image pickup device 10 according to a secondembodiment of the present invention will next be described withreference to FIGS. 25A to 27D.

As described above, the compression and decompression I/F section 105 incombination with a wavelet transform provides a method of most effectivecompression. However, not only the wavelet transform but also a Haartransform that reduces a hardware load can realize the method. At thistime, the compression and decompression I/F section 105 subjects everytwo pixels adjacent to each other in an oblique direction in image dataof a color in which pixel positions are alternately shifted from eachother to a Haar transform with two upper and lower lines adjacent toeach other as a unit.

FIGS. 25A to 25D show an example in which the compression anddecompression I/F section 105 performs a Haar transform for the doubledensity Bayer arrangement.

FIG. 25A shows the double density Bayer arrangement.

First making description for a G image, the G image has 4 k full pixels,and it therefore suffices to perform a Haar transform in place of awavelet transform in a horizontal direction and a vertical direction asin the processing described with reference to FIGS. 11A to 11D.

At this time, the compression and decompression I/F section 105 subjectsevery two pixels adjacent to each other in an oblique direction in R andB image data of RGB image data output from an image pickup elementsection 102 of the double density Bayer arrangement to a Haar transformwith two upper and lower lines adjacent to each other as a unit, andsubjects G image data to a Haar transform in the horizontal directionwith pixels of one line as a unit and a Haar transform in the verticaldirection with pixels of one column as a unit.

A Haar transform is considered to be a two-tap subband filter, and isequivalent to obtaining a sum and a difference between two pixels. Thesum corresponds to an LPF, and the difference corresponds to an HPF.

Specifically, as shown in FIG. 25B, every two pixels of a G image of4096×2160 size is subjected to a Haar transform in the horizontaldirection. Thus, as shown in FIG. 25C, a low-frequency subband G image(L) (sum) and a high-frequency subband G image (H) (difference) of2048×2160 size halved in the horizontal direction are obtained.

Next, the low-frequency subband G image (L) (sum) and the high-frequencysubband G image (H) (difference) of 2048×2160 size halved in thehorizontal direction are each subjected to a Haar transform in thevertical direction, as shown in FIG. 25C. As a result, as shown in FIG.25D, the low-frequency subband G image (L) (sum) and the high-frequencysubband G image (H) (difference) of 2048×2160 size are converted intofour kinds of subband G images of 2048×1080 size.

-   -   A subband G image (LL) of 2048×1080 size passed through a        low-pass filter (sum) in both the horizontal direction and the        vertical direction    -   A subband G image (LH) of 2048×1080 size passed through a        low-pass filter (sum) in the horizontal direction and a        high-pass filter (difference) in the vertical direction    -   A subband G image (HL) of 2048×1080 size passed through a        high-pass filter (difference) in the horizontal direction and a        low-pass filter (sum) in the vertical direction    -   A subband G image (HH) of 2048×1080 size passed through a        high-pass filter (difference) in both the horizontal direction        and the vertical direction

The low-frequency subband G image (L) (sum) and the high-frequencysubband G image (H) (difference) of 2048×2160 size are converted intothe above four kinds of subband G images of 2048×1080 size.

An R image and a B image have a checkered pattern. Thus, the R image andthe B image are subjected to a Haar transform in place of a wavelettransform with two upper and lower lines adjacent to each other as aunit.

Specifically, as shown in FIG. 25B, every two oblique pixels in a unitof two upper and lower lines adjacent to each other of R pixels issubjected to a Haar transform.

As a result of the above, the R image is converted into:

a subband R image of 2048×1080 size passed through a low-pass filter(sum); and

a subband R image of 2048×1080 size passed through a high-pass filter(difference).

Every two oblique pixels (in an oppositely oblique direction from R) ina unit of two upper and lower lines adjacent to each other of B pixelsis similarly subjected to a Haar transform to be converted into:

a subband B image of 2048×1080 size passed through a low-pass filter(sum); and

a subband B image of 2048×1080 size passed through a high-pass filter(difference).

As a result of the above, subband images of 2048×1080 size are alreadyobtained, and the size coincides with the size of the subband images ofG.

This means that subband images of uniform 2048×1080 size are obtainedfor all of RGB, that RGB full images of 2048×1080 size are obtained, andthat simple so-called color separation has been performed.

Thus, the subband images as output from the compression anddecompression I/F section 105 have the same size in the 4K full pixelsystem and the 4K double density Bayer arrangement, so that processingfrom the compression and decompression processing section 106 on downcan be made to be common processing. This is the same as in the case ofthe wavelet transform.

In FIG. 25C and FIG. 25D, the subband images are described as followsfor description of the R image and the B image having been subjected tothe Haar transforms in the oblique directions.

-   -   The subband R image of 2048×1080 size passed through the        low-pass filter (sum) is a subband R image (LL)    -   The subband R image of 2048×1080 size passed through the        high-pass filter (difference) is a subband R image (HH)    -   The subband B image of 2048×1080 size passed through the        low-pass filter (sum) is a subband B image (LL)    -   The subband B image of 2048×1080 size passed through the        high-pass filter (difference) is a subband B image (HH)

The oblique directions of the R pixels and the B pixels shown in FIG.25B are an example, and it is clear that the oblique directions arereversed when starting pixel positions of the R pixels and the B pixelsbecome different.

A case of performing a Haar transform for the oblique arrangementthree-panel system shown in FIG. 2 will next be described with referenceto FIGS. 26A to 26D.

FIG. 26A shows the arrangement of the oblique arrangement three-panelsystem.

In FIG. 26B, as described with reference to FIGS. 14A to 14D, pixelsindicated by hatching are considered to be nonexistent, and this patterncan therefore be considered to be the same as that of the R or B pixelsin the double density Bayer arrangement described above with referenceto FIGS. 25A to 25D.

A difference lies in that an R image, a B image, and a G image all havethe same pattern.

It therefore suffices to subject the R image, the B image, and the Gimage to a Haar transform in place of a wavelet transform with two upperand lower lines adjacent to each other as a unit.

At this time, the compression and decompression I/F section 105 subjectsevery two pixels adjacent to each other in an oblique direction in allof RGB image data from the image pickup element section 102 of theoblique arrangement three-panel system to a Haar transform with twoupper and lower lines adjacent to each other as a unit.

Specifically, as described with reference to FIG. 26B, every two obliquepixels in a unit of two upper and lower lines adjacent to each other issubjected to a Haar transform.

As a result of the above, the RGB images are each converted into:

a subband image of 2048×1080 size passed through a low-pass filter(sum); and

a subband image of 2048×1080 size passed through a high-pass filter(difference).

It is also shown from the above that subband images of 2048×1080 sizeare already obtained.

This means that subband images of uniform 2048×1080 size are obtainedfor all of RGB, that RGB full images of 2048×1080 size are obtained, andthat simple so-called color separation has been performed.

Thus, the subband images as output from the compression anddecompression I/F section 105 have the same size in the 4K full pixelsystem and the 4K oblique arrangement three-panel system, so thatprocessing from the compression and decompression processing section 106on down can be made to be common processing. This is the same as in thecase of the wavelet transform.

In FIG. 26C and FIG. 26D, the subband images are described as followsfor description of the Haar transforms in the oblique directions havingbeen performed.

-   -   The subband R image of 2048×1080 size passed through the        low-pass filter (sum) is a subband R image (LL)    -   The subband R image of 2048×1080 size passed through the        high-pass filter (difference) is a subband R image (HH)    -   The subband G image of 2048×1080 size passed through the        low-pass filter (sum) is a subband G image (LL)    -   The subband G image of 2048×1080 size passed through the        high-pass filter (difference) is a subband G image (HH)    -   The subband B image of 2048×1080 size passed through the        low-pass filter (sum) is a subband B image (LL)    -   The subband B image of 2048×1080 size passed through the        high-pass filter (difference) is a subband B image (HH)

The oblique directions of the pixels shown in FIG. 26B are an example,and it is clear that the oblique directions are reversed when startingpixel positions of the R pixels and the B pixels become different.

A case of performing a Haar transform for the normal Bayer arrangementshown in FIG. 2 will next be described with reference to FIGS. 27A to27D.

FIG. 27A shows the so-called normal Bayer arrangement.

In FIG. 27B, positions where real pixels are not present afterseparation into each of RGB are indicated by hatching. The R image andthe B image are already discretely reduced images of 2048×1080 size inthe separated state.

As is clear from FIG. 27B, the G image can be considered to be the sameas the R pixels or the B pixels of the double density Bayer arrangementas described with reference to FIGS. 25A to 25D.

Therefore the G image is subjected to a Haar transform in place of awavelet transform with two upper and lower lines adjacent to each otheras a unit.

At this time, the compression and decompression I/F section 105 does notapply a Haar transform to R and B image data of RGB image data from theimage pickup element section 102 of the Bayer arrangement, but subjectsevery two pixels adjacent to each other in an oblique direction in theimage data of G to a Haar transform with two upper and lower linesadjacent to each other as a unit.

Specifically, as shown in FIG. 27B, every two oblique pixels in a unitof two upper and lower lines adjacent to each other is subjected to aHaar transform.

As a result of the above, the G image is converted into:

a subband G image of 2048×1080 size passed through a low-pass filter(sum); and

a subband G image of 2048×1080 size passed through a high-pass filter(difference).

It is also shown from the above that subband images of 2048×1080 sizeare already obtained.

This means that subband images of uniform 2048×1080 size are obtainedfor all of the RGB images though the R image and the B image arediscretely reduced. Therefore, 2048×1080 RGB full images are obtained,and simple so-called color separation has been performed.

Thus, the subband images as output from the compression anddecompression I/F section 105 have the same size in the 4K full pixelsystem and the 4K normal Bayer arrangement, so that processing from thecompression and decompression processing section 106 on down can be madeto be common processing. This is the same as in the case of the wavelettransform.

FIG. 27C and FIG. 27D, the subband images are described as follows fordescription of the G image having been subjected to the Haar transformin the oblique direction.

-   -   The R image of 2048×1080 size is a subband R image (LL)    -   The subband G image of 2048×1080 size passed through the        low-pass filter (sum) is a subband G image (LL)    -   The subband G image of 2048×1080 size passed through the        high-pass filter (difference) is a subband G image (HH)    -   The B image of 2048×1080 size is a subband B image (LL)

The oblique direction of the G pixels shown in FIG. 27B are an example,and it is clear that the oblique direction is reversed when a startingpixel position of G becomes different.

According to the second embodiment described above, the compression anddecompression I/F section 105 can be realized by not only a wavelettransform but also a Haar transform for simpler hardware. Thus, as inthe first embodiment, subsequent processing can be performed as parallelprocessing. In addition, because color separation can be made simply,the cost of manufacturing the device can be reduced.

It is to be noted that while wavelet transforms have been described withreference to the 5×3 reversible wavelet transform coefficients ofJPEG2000, it is clear that wavelet transforms can also be realized with9×7 irreversible transform coefficients and other transformcoefficients, and are not limited to a 5×3 reversible wavelet transformor a Haar transform.

<3. Examples of Modification>

It is to be noted that the processing of the compression anddecompression I/F section 105 and the processing of the compression anddecompression processing section 106 described above can be carried outnot only by hardware but also by software. When the series of processesis to be carried out by software, a program constituting the softwarecan be executed by a computer incorporated in dedicated hardware or acomputer on which programs for performing various functions areinstalled. For example, it suffices to install and execute a programconstituting desired software on a general-purpose personal computer orthe like.

In addition, a recording medium on which the program code of softwarefor realizing the functions of the above-described embodiments isrecorded may be supplied to a system or a device. In addition, it isneedless to say that the functions are realized when a computer (or acontrolling device such as a CPU or the like) of the system or thedevice reads and executes the program code stored on the recordingmedium.

Usable as the recording medium for supplying the program code in thiscase is for example a flexible disk, a hard disk, an optical disk, amagneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatilememory card, a ROM or the like.

The functions of the above-described embodiments are realized byexecuting the program code read by the computer. In addition, an OSoperating on the computer or the like performs a part or all of actualprocessing on the basis of directions of the program code. A case wherethe functions of the above-described embodiments are realized by theprocessing is also included.

In addition, while the present invention is applied to an image pickupdevice in the above embodiments, the present invention is alsoapplicable for compression coding of RAW data obtained from image pickupelements of the double density Bayer arrangement, the obliquearrangement three-panel system, and the Bayer arrangement in devicesother than image pickup devices.

In addition, the present invention is similarly applicable forcompression coding of RAW data obtained from an image pickup element ofa pixel arrangement that is other than the double density Bayerarrangement, the oblique arrangement three-panel system, or the Bayerarrangement and in which pixel arrangement pixel positions of at leastone of three primary colors are alternately shifted from each other in ahorizontal direction or a vertical direction.

In addition, the present invention is not limited to the above-describedembodiments, and various other examples of application and various otherexamples of modification can of course be adopted without departing fromthe spirit of the present invention described in claims.

The present application contains subject matter related to thatdisclosed in Japanese Priority Patent Application JP 2009-288074 filedin the Japan Patent Office on Dec. 18, 2009, the entire content of whichis hereby incorporated by reference.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factor in so far as they arewithin the scope of the appended claims or the equivalents thereof.

What is claimed is:
 1. An image processing device comprising: a subbanddividing section configured to perform subband division of color imagedata included in pickup image data output from an image pickup element,the pickup image data having pixel positions of respective primary colorcomponents that are alternately shifted from each other in a horizontaldirection or a vertical direction, the subband division producingrespective subband images of respective primary color components of thepickup image data, the primary color component images havingcombinations of low and high frequencies in horizontal and verticaldirections of the image, the pixels of the respective primary colorcomponents that are disposed in two adjacent upper and lower horizontallines or the pixels that are disposed in two adjacent left and rightvertical columns comprising a unit, the subband division not beingperformed on color image data of a primary color component whose pixelpositions are not alternately shifted from each other; and a compressioncoding section configured to perform compression-coding of the colorimage data output from said subband dividing section in parallel foreach of the subband images and for each of said three primary colors. 2.An image processing method comprising the steps of: subband dividingcolor image data included in pickup image data output from an imagepickup element, the pickup image data having pixel positions ofrespective primary color components that are alternately shifted fromeach other in a horizontal direction or a vertical direction, thesubband division producing respective subband images of respectiveprimary color components of the pickup image data, the primary colorcomponent images having combinations of low and high frequencies inhorizontal and vertical directions of the image, with the pixels of therespective primary color components that are disposed in two adjacentupper and lower horizontal lines adjacent or the pixels that aredisposed in two adjacent left and right vertical columns comprising aunit, the subband division not being performed on color image data of aprimary color component whose pixel positions are not alternatelyshifted from each other; and compression-coding the color image dataresulting from the subband division in parallel for each of the subbandimages and for each of said three primary colors.
 3. An image processingdevice comprising: subband dividing means for performing subbanddivision of color image data included in pickup image data output froman image pickup element, the pickup image data having pixel positions ofrespective primary color components that are alternately shifted fromeach other in a horizontal direction or a vertical direction, thesubband division producing respective subband images of respectiveprimary color components of the pickup image data, the primary colorcomponent images having combinations of low and high frequencies inhorizontal and vertical directions of the image, the pixels of therespective primary color components that are disposed in two adjacentupper and lower horizontal lines or the pixels that are disposed in twoadjacent left and right vertical columns comprising a unit, the subbanddivision not being performed on color image data of a primary colorcomponent whose pixel positions are not alternately shifted from eachother; and compression coding means for performing compression-coding ofthe color image data output from said subband dividing means in parallelfor each of the subband images and for each of said three primarycolors.